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Research  Issues 


•  Investigate  modeling  of  reflexive  reasoning  in 
the  CIC  task. 

•  Does  metacog nitive  /  reflective  skill  improve 
the  efficiency  and  accuracy  of  reflexive 
iearning? 

•  How  is  metacognitive  skill  itself  learned? 


Initial  machine  learning  study  addresses  the 
second  issue. 


Research  Overview 


1 .  System  Concept 

-  Integrating  adaptable  rapid  parallel  reflexive  reasoning  with 
adaptable  executive  processes  for  critiquing  and  correcting 

2.  SHRUTI  Enhancements  (Shastri) 

3.  Hybrid  System  Development 

-  Knowledge  Base  Encoding 

-  Metacog n it ive  System 

-  Adaptive  Critic  architecture 

4.  Machine  Learning  Study 

-  Learning  abductive  reasoning  by  localized  backpropagation 

-  Effect  of  a  metacognitive  rule  on  domain  learning 

5.  Human  Learning  Studies 

6.  Future  Plans 


1.  Hybrid  Architecture  for 
Metacognitive  Learning 
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An  Integration  of  Three  Research  Areas 


*  Recognition  /  Metacognition  modei  of  criticai 
thinking  in  decision  making. 

*  SHRUTI,  a  connectionist  model  of  rapid 
parailel  reasoning  with  dynamic  variabie 
binding 

*  Adaptive  Critics,  a  connectionist  modei  of 
behavior  learning. 
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'  How  Does  the  Architecture  Relate 
These  Three  Components? 


Recognition  Metacognition 

Adaptive  Critic  Adaptive  Critic 


Critic 

Critic 

Situation  Model 
(SHRUTI) 

Model  of  the 
Situation  Model 

Action  Net 

Action  Net 

Implements 
Recognition  / 
Metacoqnition 
Model 
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Functionality  of  Components 


Recognition 
Adaptive  Critic 


Ensure  that  selected 
action  is  best  for  the 
given  situation  model 

SHRUTI:  Represent  and 
reflexively  reason  about 
situation  (e.g., 
kinematics,  EW,  intent). 


Metacognitive  Adaptive 
Critic  (implements  R/M 
Model) 


Ensure  that  critical 
thinking  occurs  only 
when  time,  stakes, 

&  uncertainty  justify  it. 

Identify  arguments, 
gaps,  conflict,  & 
unreliability  in 
situation  model 


Critic 


Model 


Shift  attention  and 
clamp  truth  values  Action 

in  situation  model.  Net 

Inhibit  immediate  action 


Take  action 
(engage, 
warn,  etc.) 


Hybrid  System: 
Learning  in  Adaptive  Critics 


\ 


Critic  iearns  to  predict  cumuiative 
expected  reinforcement  associated 
with  state  or  state  and  action 


Model  learns  good  predictions  of 
vaiued  events. 


Action  Net  iearns  to  reflexively  take 
actions  that  maximize  expected  value 
of  outcome. 


Summary  of  Hybrid  Concept 


*  Hybrid  model  captures  both  reflexive 
(recognitional)  processes  and  metacognition 
(critical  thinking) 

*  The  combination  of  these  is  what  enables 
decision  makers  both  to  exploit  their 
experience  and  to  go  beyond  it 

*  Metacognition  leverages  existing  knowledge 
in  novel  and  uncertain  situations 

*  Metacognition  increases  efficiency  of  domain 
learning 


Knowledge  Base  Encoding  In 

SHRUTI 


Sources 

-  Critical  incident  interviews  with  active-duty  navai  officers 

-  Participants  of  experimentai  triais  at  Surface  Warfare 
Officers  Schooi  and  Navai  Post  Graduate  Schooi 


Intent  Story  Structures 


SHRUTI  Enhancements 


Negation 

Abductive  reasoning 
Backpropagation 
Many  others 


Metacognitive  Model  in  Hybrid 

System 


•  Bound  on  length  of  reasoning  chain  and  limit  on 
number  of  objects  in  SHRUTi  means  aii  reievant 
information  in  long  term  knowledge  base  may  not  be 
retrieved  in  first  cycie  of  activation 

*  Metacognitive  modei  provides  mechanism  for 
managing  recognitional  processing  by  shifting 
attention  and  ciamping  values 


*  Metacognitive  critic  balances  advantages  of  thinking 
more  versus  risks  of  delaying  action 
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Metacognitive  Attention  Shifting 


approproiate-platform- 
f  or- attack 

(track, asset2,country1 ) 


wants-to-ch  allonge 
(countryl  ,country2) 


owns 


(country2, asset) 


in-danger(assetl) 


Notice  conflict 
at  hostile 
intent 


owns 

(countryl , track) 

owns 

(countryl  ,asset1) 


hostile-intent 
(track, asset2) 


-NOT- 


nonhostile- 
intent( track,  asset2) 


closing-on 

(track,asset2) 


Shift  attention 
to  non-hostile 
intent 


approproiate-plattorm-tor- 

protection 

(track,asset1 , countryl ) 


▼ 

intends-to- 
protect(track,asset1 ) 


closing-on 

(track,asset1 


Activation 
spreads  to 
arguments 
supporting 
Intent  to 
Protect 
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Machine  Demonstration 


Reflexive  performance 

*  hostile  intent  +  675,  - 169,  conflict  114 

•  intent  to  protect  +  0,  -  394,  confiict  0 

*  less  experienced  officer  might  engage 

Performance  with  metacognitive  skill 

*  hostile  intent  +  733,  - 168,  confiict  123 

•  intent  to  protect  +  415.  -  0,  confiict  0 

•  more  experienced  officer  might  wait 


4.  Machine  Learning  Study 


•  Procedure 

-  Generate  scenarios  according  to  “reai-worid” 
probabiiities  in  event  tree 

-  Randomiy  divide  scenarios  into  training  and  test  sets 

-  Use  backpropagation  to  adapt  weights  in  training 
scenarios 

-  in  refiexive  condition,  train  each  scenario  for  50  cycies 

-  in  metacognitive  condition,  insert  “hint”  to  query  non- 
hostiie  intent  midway  through  training 


*  Dependent  Variables 

-  Changes  in  weights  (i.e.,  knowiedge  base  adaptation) 

-  Probabiiity-weighted  sum  of  squared  errors  in  test 
scenarios 

-  Performance  in  standard  scenario  (Korea) 
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Scenario  Generation: 
Probabiiistic  Event  Tree 


in-danger 


wants-to- 

challenge 


“in-danger- 


~  wants-to- 
challenge 


in-danger 


“in-danger 


appropriate- 

platform 


“appropriate- 

platform 


appropriate- 

platform 


-appropriate- 

platform 


appropriate- 

platform 


“appropriate-_| 
platform 


appropriate- 

platform 


-appropriate- 

platform 


hostile-intent 
intent-to-protect 
— other  intent  — 
— hostile-intent 
intent-to-protect 
- other - 


hostile-intent 
intent-to-protect 
- other - 


-hostile-intent 

intent-to-protect 

- other - 

— hostile-intent 


intent-to-protect 
— other  intent  — 
— hostile-intent— 


intent-to-protect 

- other - 

— hostile-intent 


-intent-to-protect 

- other - 

— hostile-intent 
intent-to-protect- 
- other - 
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closing-on 


hostile-intent 


intends-to- 

protect 


~  closing-on 


closing-on — 


~  closing-on 


high  speed 


low  altitude 


~  low  altitudeH 


low  altitude 


~  low  altitude 


~  high  speed 
high  speed- 

~  high  speed 
high  speed 

~  high  speed 
high  speed 

~  high  speed 
high  speed 


low  altitude 


~  high  speed 
-high  speed- 


~  low  altitude- 


low  altitude 


~  low  altitude 


—  high  speed 
— high  speed- 

—  ~  high  speed 
— high  speed 

—  high  speed 


Demonstration:  Effect  of 
Metacognitive  “Hint”  During  Training 


Reflexive  Training:  Correction  applied  to  hostiie  intent 

*  Change  in  weight  on  rule  for  hostile  intent  from  900 
to  750 


Training  with  metacognition:  Correction  as  above, 
pius  “hint”  to  think  about  non-hostile  intents 

*  Change  in  weight  on  rule  for  hostile  intent  from  900 
to  750 


•  Increase  in  “prior  beiief”  in  likeiihood  of  non-hostiie 
intent  from  50  to  64 


Decrease  in  “prior  belief” 
intent  from  50  to  49 


in  likeiihood  of  hostiie 


utility  Function  for  Reflexive  Critic 


•  U(mil, intent, act)  +  U(attacked,mil)  +  U(ROEviol) 

*  Illustrative  values 


U(mil, intent, act) 


Military 

Civilian 

Hostiie 

intent 

Engage  20 
Not  eng  0 

Engage  10 
Not  eng  0 

No  hostiie 
intent 

Engage  -30 
Not  eng  0 

Engage  -100 
Not  eng  0 

□(attacked, mil) 


Military 

Civilian 

Attacked  -150 
Not  attacked  0 

Attacked  -75 
Not  attacked  0 

utility  Function  for  Metacognitive 

Critic 

•  Change  in  payoffs  due  to  change  in  actions 
due  to  change  in  beiiefs 

•  Involves  comparison  of  future  utility  expected 
given  critical  thinking  versus  utility  of 
outcome  that  would  occur  if  officer  acted 
immediateiy  on  current  best  model 


Decision  to  accept  vs.  question  recognitionai  response 


-  Confidence  in  recognitionai  response 

-  Cost  of  delay 

-  Cost  of  error  in  acting  on  current  best  response 


High  stakes,  low  time  stress  Decreasing  stakes,  low  time  stress 


Handling  Tracks:  Complementary 

Schemas 


ROE-EXECUTION  SCHEMA 


Start  with  tracks  within  20 
nm  of  ownship  and  move 
out  in  expanding  circles  to 
30,  40,  &  50  nm 

Query  positive  support  for 
non-hostile(track,Ownship) 


Implement  three  levels  of 
warning,  locking  on,  and 
engagement 


THREAT-ASSESSMENT 

SCHEMA 


Start  with  tracks  within  their 
weapons  range  of  own  ship, 
move  out  to  tracks  that  can 
detect  own  ship 


Query  positive  support  for 
hostiie-intent(track,Ownship) 


May  iead  to  engagement 
outside  narrow  ROE  criteria, 
or  non-engagement  when 
inside  ROE  criteria 
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Distinctive  Aspects  of  Modei 

Predictions 


*  Reasoning  depends  critically  on  dynamic  binding 

-  Different  countries  own  the  two  piatforms  in  attack  situation 
(attacking  and  attacked) 

-  Same  country  owns  two  piatforms  in  protection  situation 

*  Includes  background  knowledge  underlying  “base 
rates” 

*  Applies  to  situations  where  all  long-term  knowledge  is 
not  immediately  accessible  due  to  cognitive  limits 

*  Model  predicts  shifts  of  attention  and  resulting 
changes  in  belief 

*  Uncertainty  is  treated  as  a  problem  to  be  solved  rather 
than  averaged  away 


Effects  of  Metacognition  on  Long 
Term  Memory  Structure 

*  General  metacognitive  skills 

-  When  and  where  to  shift  attention  based  on  confiict, 
gaps,  unreiiabiiity 

*  Domain-specific  metacognitive  skiils 

-  Which  predicates  to  scrutinize  when  troubie  arises 

*  Speeds  compiiation  of  domain  knowiedge 

-  Extends  reach  of  backpropagation  iearning 

-  Constructs  new  iinks 
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'  A  Training  Tool: 

The  STEP  Framework  for  Critical 

Thinking 


1 .  Stories 

Create  a  story  based 
on  your  assessment. 
Identify  and  fill  gaps 
in  the  story 


3.  Evaluate 


Identify  and  evaluate 

assumptions  in  the  story. 
Identify  and  evaluate 

alternative  assessments. 


2.  T EST 

Identify  and  explain 
data  that  conflict 
with  the  story 


4.  Plan 

Develop  contingency 

plans  for  the  weakest 
assumptions  in  your 
story. 
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Scenarios  and  Simulation 


*  5  Scenarios  developed  by  NCCOSC  as  part  of 
the  TADMUS  program. 

•  Enhanced  by  CTI  to  emphasize  features 
relevant  to  critical  thinking 

-  Information  often  matches  no  single  pattern 

-  Scenarios  extended  so  that  tracks  come  closer  to  own 
ship 
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NPS  Study 


Participants 

*  35  graduate  students  at  the  Navai  Post 
Graduate  School 

•  Median  years  of  service  =  9.5  years 

Experimental  Design 

Scenario  Order  x  Test  (Pre  vs  Post) 

Procedure 

*  Total  of  five  2-hour  sessions,  spread  over  two 
weeks 

•  Four  hours  of  training 


Testing,  Elaborating,  &  Evaluating 

Stories 


Trained  participants  identified  more  confiicting 
evidence 

-  NPS:  58%  increase  (p<.001) 

Trained  officers  identified  more  assumptions 
underiying  confiicting  evidence 

-  NPS:  27%  increase  (p<.001 

Trained  participants  generated  more  alternative 
assessments 

-  NPS:  41%  increase  (p<.001) 

Training  significantly  increased  agreement  with 
SME  in  two  of  the  four  test  scenarios  --  one  at 
both  NPS  and  SWOS  (p  =  .034,  p  =  .013;  effect  ns 
in  the  other  scenarios) 


6.  Future  Plans 


Reinforcement  learning 

-  Domain 

-  Metacognitive 

Testing 

SHRUTI  enhancements 


X 


Machine  Exploration  of  Theory 

*  Determine  relative  importance  of  different 
metacognitive  capabilities 

-  Metacognitive  actions:  Shift  focus  versus  ciamp  truth  vaiues 

-  Metacognitive  modei  eiements:  identify  arguments,  gaps, 
confiict,  &/or  unreiiabiiity  in  knowiedge  base 

-  Simuiate  and  examine  vaiue  of  specific  techniques,  such  as 
devii’s  advocate  (crystai  baii) 

*  Determine  relative  importance  of  general 
metacognitive  rules  versus  domain-specific 
metacognitive  rules 

»  identifying  arguments,  gaps,  confiict,  unreiiabiiity 

»  Which  arguments  are  examined  first?  Pattern  and 
sequence  of  corrective  actions.  Standards  for  judging 
reiiabiiity.  Methods  for  assessing  risk  and  stakes 


Human  &  Machine  Testing 


Specific  predictions  of  Hybrid  Model 

•  Benefits  of  metacognitive  skill  depend  on 
limitations  in  length  of  inferential  chains  in 
reflexive  reasoning 

•  Benefits  of  metacognitive  skill  increase  with 
the  number  of  objects  to  be  reasoned  about 

*  Order  of  generation  of  arguments  and 
explanations  corresponds  to  length  of 
inferential  chains 

*  Retrospective  reinterpretation  of  previously 
considered  evidence  occurs  as  a  result  of 
new  conflicting  data 


Human  &  Machine  Testing 


*  Compare  alternative  training  interventions 

-  Practice 

»  Through  practice,  strategies  shouid  become  more  denseiy 
connected  to  KB  and  more  fiuidiy  executed  (i.e.,  “compiied”) 

-  Hints  &  feedback  #1 :  Direct  attention 

»  What  is  track  #7031  doing? 

»  What  do  you  think  about  the  intermittent  iFF  response? 

»  is  that  track  a  more  iucrative  target  than  own  ship? 

-  Hints  &  feedback  #2:  Ciamp  nodes  true  or  faise 

»  if  the  track  is  hostiie,  what  wouid  you  expect  to  see? 

»  What  eise  couid  that  observation  mean? 

»  Suppose  you  are  wrong  in  your  assessment.  Expiain  how  that 
couid  be. 
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The  Recognitional  Model:  SHRUTI 


*  Rapid  reflexive  inference 

-  Time  independent  of  size  of  long  term  knowledge  base 

-  Space  linear  in  size  of  long  term  knowledge  base 

•  Neurally  plausible  dynamic  variable  binding 

-  Goes  beyond  association  of  propositions  or  features  (e,g.,  in 
schemas) 

-  Supports  complex  relational  reasoning  among  multiple  objects, 
features,  and  relations 


•  Implies  limitations  on  reflexive  reasoning 

-  Limits  on  number  of  objects  &  length  of  reasoning/retrieval 
chain,  not  on  number  of  active  facts  or  rules 


Provides  context  in  which  metacognitive  skills  may  improve 
both  performance  and  learning 


Major  SHRUTI  Enhancements 

Completed 

Negation 

-  Support  for  negative  as  well  as  positive  knowledge.  Explains 
how  agent  can  hold  inconsistent  beliefs,  and  become  aware  of 
contradiction  only  when  beliefs  are  within  a  certain  inferential 
distance  of  each  other 

Abductive  reasoning 

-  Inference  from  occurrence  of  an  effect  to  its  likely  cause; 
combination  of  evidence  regarding  a  cause 

Derivative  computation  to  support  back 

propagation  /  reinforcement  iearning 
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(  SHRUTI  Enhancements  Completed 


•  Summary  facts 

-  Numerically  represent  strength  of  belief  in  a  fact 

-  Strength  of  belief  affected  by  types  of  role  fillers,  e.g.,  a  particular 
type  of  aircraft  is  more  iikely  to  attack  cruiser  than  other  types 

•  Reduced  restrictions  on  ruies 

-  Regarding  multiple  occurrences  of  same  variable  in  antecedent  or 
consequent  but  not  both 

•  Continuous  attribute  values 

-  Representation  of  scalar  quantities  such  as  speed  and  altitude 

•  Action  schemas 


-  Procedural  sequences 


*  Mutual  exclusion  in  type  hierarchy 

•  Ability  to  save  snapshots  of  network  weights  and 
activity  patterns 


Metacognitive  Model 


Representations  of 

*  Attended  arguments  in  refiexive  model  (conclusion- 
grounds  pairs) 

•  Different  types  of  uncertainty  associated  with 
arguments  in  reflexive  model 

-  Incompleteness  The  grounds  of  an  argument  activate  neither  +  nor  - 
collector  of  conclusion 

-  Conflict:  Groundsi  activates  +  and  grounds2  activates  -  collector 
for  conclusion  —  provided  that  query  involves  only  instantiated  or 
universally  quantified  variables 

-  Unreliability:  Grounds  activate  either  +  or  -  conclusion  and  not  both, 
but  activation  is  changeable  due  to  as  yet  unconsidered  arguments 

V  1 


Metacognitive  Action  Net 


Basic  metacognitive  actions 

*  Shift  attention  /  query  predicates  piaced  in 
intermediate  memory  by  reflexive  model 

-  Training  anaiog:  “hints”  to  think  about  a  possibiiity 

*  Clamp  predicates  true  or  false 

-  Training  anaiog:  “hints”  to  imagine  your  conciusion  is  wrong,  hints 
to  iook  for  aiternative  expianations  or  predictions 

*  Inhibit  actions  of  reflexive  system 

-  Training  anaiog: 


Aspects  of  Knowledge  Base 


Active  schemas  for  handling  tracks 

-  Scan  tracks  according  to  ROE  and  doctrine  based 
responding 

-  Assess  tracks  for  threat  status 

Intent  Story  Structures 

-  Inciudes  overaii  situation  (goais,  capabiiities, 
opportunities)  and  preparatory  actions  (e.g.,  iocaiization, 
track  kinematics  and  EW) 

Structures  for  defeasible  reasoning 

-  Allows  for  rapid  initiai  inference,  then  possibie 
withdrawai  of  conciusion  on  further  examination 


Reflexive  Actions 


*  External  response 

-  engage 

-  jam 

-  launch  chaff 

-  vector  CAP 

-  maneuver 

-  illuminate 

-  warn  (levels  1 ,2,3) 

-  query 

-  offer  help 


•  Internal  response 

-  activate  AAW  auto 

-  prepare  defenses 

-  get  targeting  solution 

-  request  assets 

-  designate  hostile 

-  set  alert  level 

-  broadcast  to  battle  force 

-  notify  superior(s) 

-  monitor 

-  review  /  study  /  recall 
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Types  of  Stories 


•  Attack 

•  Targeting 

•  Harass 

•  Provoke 

•  Testing  response 

•  Reconnaissance 

•  Patroi 

•  Exercise 

•  Training 

•  Friendiy 

•  Commair 

•  Search  &  rescue 
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Reinforcement  Learning 
Accompiishments 

Specification  of  utility  function  for 

Recognition  Critic 

-  Represents  expected  vaiue  given  the  recognition  modei 
and  seiected  action 

-  Supports  iearning  the  best  overt  action  given  the 
scenario 

Specification  of  utility  function  for 

Metacognitive  Critic 

-  Represents  expected  vaiue  given  the  recognition  modei, 
the  seiected  action,  and  the  uncertainty  in  the 
recognition  modei 

-  Supports  iearning  the  best  metacognitive  action  given 
the  recognition  modei 

Benchmark  model  of  attention  shifting 


Backpropagated  Adaptive  Critic 

(BAG) 


X(t),  R(t) 

Situation  at  time  t 


Critic 


J*(t+1 )  Estimated  totai 
future  utiiity  as  of  t+1 


R(t+1)  Situation 
predicted  for  time  t+1 


U(t+1)  Utiiity  of 
situation  predicted  for 
time  t+1 


Modei 


^(t+1) 


Observable 


predicted  for  time  t+1 


U(t)  Actions  at  time  t 


Action 
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Critic  Learning 


Situation  at  time  t+1 


R(t+1) 


Estimated  totai  future 
utiiity  as  of  t+1 


Critic 


Situation  at  time  t 


Estimated  totai  future 
utiiity  as  of  t 


Utiiity  of  situation 
predicted  for  time  t+1 


R(t) 


Critic 


Adjust  critic  weights  so  that 
J*(t+1)  +  U(t+1)  =  J*(t) 


Dynamic  Programming 


•  General,  exact,  and  efficient 

•  Maximizes  utility  over  time. 

•  Can  control  multiple  variables. 

•  Works  in  noisy  and  nonlinear  environments. 


Model  of 
Reality  (F) 


Utility 

Function  (U) 


I  Y 


DYNAMIC  PROGRAMMING 

1 

Y 


Secondary  or  Strategic 
Utility  Function  (J) 


Benchmark  Model  of  Attention 

Shifting 

Based  on  decision  theoretic  Value  of 
Information  concept,  extended  to  the  value  of 
further  attention,  or  thinking  about  a  problem 

Generalized  to  case  where  possible 
outcomes  of  observations  /  thinking  are 
unspecified  or  only  vaguely  known 

Manages  thinking  process 

-  Avaiiabie  time  =  costs  of  deiay  in  thinking  more 

-  Stakes  =  costs  of  potentiai  errors  in  acting  on  current 
recognition 

-  Uncertainty  =  match  of  situation  to  recognitionai 
patterns,  and  specific  probiems  (gaps,  confiicts, 
unreiiabiiity) 


C  T 


Scenario  Delta 


[T”|^ —  Cessna  (suicide  ac?) 


m  m  rn  neios  inbound:  0min/42nm 


F4  EW  near^ 
Cessna:  2min 


/•t . '  \ 

Coastline  ^\l 

EW:  7min  — 


EW-Smin’ 

Helo  change  course: 
\  \  13min  /  10nm  ( 


-T'./EW:  7  &  10min 


Helos'  CPA:  12min  /  10nm 


Melo  change 
cc'urse:  16min  / 
_  /  6nm 


Flagshif: 


EW:  13min 


F4:  Omin  /  ^7nm 


C2  VIDs  Cessna,  Cessna  dives: 
16min  /  5nm 


Commair 


P3  EW  &  turn:1 5min  /  43nm 


P3  turns  inbound: 
12min  /  60nm 
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Scenario  Tango 


Merchants 


Helo  GPA:  10  min  /  In 


Helo  departs:  5min  /  lOnm 


Coastline  Commair: 

■; - Imin/ 

43nm 


/>tW:  3min  / 
28nm 
EW:  8min  /  24nm 


■13min  / 18  nm 


EW:  15min 


F4s  divert^  7min  /  6nm 


Helo  lands:  15min  /  lOnm 


Merchant 


Scenario  Bravo 


Scenario  Charlie 


Commair 

n  / 


25min 


F4s  on  touch- 
and-go:  5min,  77nm 


Commair  'EW:10min 

Area  of  operations 


Helo  departs  island:  Omin  /  38nm 


P3  &  F4s  CPA:  28min7  17nm  ; 
Missile  or  zoomer  inbound:  28min  / 

/  '  □ 


Coastline 


O  \ 


Held  SOS:  22min  /  6nm  •' 

-A.  >► 


Commair  CPA's: 
12min  &  20  min 


Commair  CPA's: 
5min  &  20  min 


Boghammer  (tPj 
^  18min/14i/im/ 


Patrol  craft 


Commair 


ommair  CPA:/ 
16min  /  1nm/ 


EWP3CPA:9min/12nm  Commair:  10min 


Mirage  CPA;  21min  /  6nm 


Coitimair  CPA: 
1 0itiin  /  Onm 


Faicoii  orbits  Tisdaie: 
11min/9nm 


P3:  1min  /48nm 


Area  of  operations 


Coastiine 


Commair: 


Faicon  EW  &  turns  to 
Tisdaie:  9min  /  9nm 


Faicon:  Omin  / 


5.  Behavioral  Testing 


Developed  efficient  measures  of  critical 
thinking  processes 

Developed  methods  for  manipulating 
metacognitive  processing  through  training 

Coilected  baseline  data  on  human 
performance  for  comparison  to  machine 

Tested  hypotheses  regarding  track 
prioritization  by  Navai  officers 


Training  Metacognitive  Skiiis  for 

Decision  Making 

Four  segments 

•  STEP:  A  Critical  Thinking  Method 

•  The  Hostile  Intent  template 

•  The  Crystal  Ball  technique 

•  The  Quick  Test:  When  to  think  more 

Each  segment  includes: 

•  Presentation  by  instructor 

•  Discussion 

•  Exercises  with  simulated  scenarios 

»  Continuous  reai-time  probes,  discussion,  and 
feedback  with  instructor 


C  T  I  I 

1  ^Questions  at  Each  Break  in  Test 

Scenarios 

1  Assess  intent  of  designated  tract*  and  give 
reasons 

2  Give  other  possibie  intents  and  confidence  in 
each 

3  Seiect  a  disbelieved  intent  and  give  reasons  it 
couid  be  true 

4  Identify  evidence  that  confiicts  with  a 
stipulated  intent.  Try  to  expiain  the  evidence. 

-  May  or  may  not  be  the  subject’s  favored  intent  in 
question  1 

5  What  actions  wouid  you  take? 


*  Designated  track  not  known  by  participants 
before  break 


Filling  Gaps  in  Stories 


•  Trained  participants  generated  more 
arguments  for  a  hypothesis 

-  30%  increase  (p=.001) 

•  Training  increased  reasoning  based  on 
factors  other  than  track  kinematics:  i.e.,  prior 
situation  &  motives,  opportunities,  capability, 
localization 


Training  increased  arguments  based  on 
predicted  actions 


Arguments 
Generated 
(ave.  per 
hypothesis) 


4.5 
4 

3.5 
3 

2.5 
2 

1.5 
1 

0.5 

0 


V 


Pretest 

mean 


Posttest 

mean 


X 


Potential  Tests  of 
Metacognitive  Learning 

Handling  conflicting  data 

*  Present  sequences  of  cues  supporting 
hostile  or  friendly  intent 

-  Group  1 :  H1  H2  H3  F1  F2  F3 

-  Group  2:  F1  F2  F3  H1  H2  H3 

•  “Confirmation  bias” 

-  Group  1  interprets  F1  as  hostile 

-  Group  2  interprets  H1  as  friendly 

•  R  /  M  model  accounts  for  this  as  a  form  of 
explanation-based  reasoning 

-  Reinterpret  cues  in  order  to  construct  a  single,  coherent 
story  (critique  reliability  of  conflicting  arguments) 

*  According  to  R  /  M,  if  too  many  unreliable 
explanations  are  required,  story  is  rejected 


L^t,! 

'  Handling  Conflicting  Data 

Independent  variables 
•  Order  of  cues 


•  Number  of  conflicting  cues  before  engagement  is 
possible 

-  The  more  explanations  required,  the  more  likely  story  is  to  be 
rejected 

*  Susceptibility  of  F  cues  or  H  cues  to  a  single 
explanation  (e.g.,  same  source,  change  in  tactics  or 
conditions  over  time) 

-  If  conflicting  cues  are  subject  to  a  single  explanation,  more  are 
required  to  cause  change  of  mind 


*  Sequentiai  vs.  simuitaneous  presentation  of 
conflicting  cues 


-  If  conflicting  cues  are  sequential,  more  ability  to  construct 
V  explanations  and  less  ability  to  evaluate  overall  reliability  of  story 


xperience  in  the  domain 
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Metacognitively  Driven  Domain 

Learning 

*  Test  learning  about  cues  in  a  subsequent 
closely  related  scenario 

-  Group  1  more  iikeiy  to  interpret  F1  as  hostiie 

-  Group  2  more  iikeiy  to  interpret  H1  as  friendiy 

*  Contrast  training  with  novel  problems  versus 
training  with  a  consistent  problem  set 

*  Training  with  novel  problems  builds 
metacognitive  skill 


